Method for remote control, system using the same method, and computer 
program product performing the same method 



FIELD OF THE INVENTION 

5 The present invention relates to a remote control data processing 

method in a remote control system including a client computer, a host computer, 
and a network for linking them, a system using the method, and a computer 
program product performing the method. 

10 BACKGROUND OF THE INVENTION 

As communication networks have recently expanded and computer 
systems mounting easyto-use graphical user interface are developed, remote 
control systems for controlling electronic appUances at distant places through 
the network are being widely distributed. 
15 One of the problems of remote control by network is that it takes a 

certain time to transmit a remote control signal over a long distance. As a 
result, a time delay occurs between the local operation and operation at the 
distant place. 

The delay time in the network becomes a serious problem in the case of 
20 composite operations which are composed of plural operations. In such 
composite operations, the interval between one operation and another is an 
important factor for judging what operation has been done. For example, 
suppose a computer program is opened by double-clicking an icon on the screen 
of a computer in a distant place. 
25 An example of a problem in remote control which is caused by a delay 

time in the network is explained below. 

Fig. 8 is a configuration of a remote control system. 
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In Fig. 8, the remote control system contains a client computer 2, a host 
computer 1 which is controlled by the client computer 2 from a distant place, 
and a network 3 Unking these computers. 

The host computer 1 comprises an information processing unit 11, a 
5 display unit 12, a communication unit 13, and a pointing device 14. The 
display unit 12 displays the image information that is processed in the 
information processing unit 11. The communication unit 13 exchanges 
information with the network 3. The pointing device 14 manipulates the host 
computer 1. 

10 The cHent computer 2 comprises an information processing unit 21, a 

display unit 22, a communication unit 23, and a pointing device 9. The display 
unit 22 displays the image information that is processed in the information 
processing unit 21. The communication unit 23 exchanges information with 
the network 3. The pointing device 9 manipulates the client computer 2. 

15 The pointing devices 14, 9 are a mouse, a digitizer, and other input 

devices. 

The pointing devices 14, 9 have on/off buttons for moving the pointer so 
as to indicate a position on the screen of the respective display units 12, 22, and 
to select the object that is indicated by the pointer by inputting coordinates 
20 data. 

The user can cHck the on/off button twice quickly without moving the 
pointing device (without updating the device coordinates data), which is called a 
double click. In this case, if the interval between the first cUck and second 
chck is long, this operation is not recognized as a double cUck. 
25 The network 3 is a wired or wireless network such as a local area 

network (LAN) or a wide area network (WAN). 

The operation of the remote control system will now be explained. 
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The host computer (hereinafter called host PC) 1 transmits the image 
information that is output in the display unit 12 to the communication unit 23 
of the cHent computer (hereinafter called client PC) 2 from the communication 
unit 13 through the network 3. At this time, the cHent computer 2 processes 
5 the image information that is sent from the host computer 1 in the information 
processing unit 21, and outputs the processed image information to the display 
unit 22. 

When the pointing device 9 is manipulated in the client PC 2, this 
operation information is sent to the communication unit 13 from the 
10 communication unit 23 through the network 3. The host PC 1 processes the 
operation information that is sent from the cHent PC 2 as the operation 
information by manipulation of the pointing device 14. 

The host PC 1, when the image information that is output in the display 
unit 12 is changed by manipulation of the pointing device 14, sends the 
15 differential image information before and after the change from the 
communication unit 13 to the communication unit 23 of the client PC 2. 

The client PC 2 updates the image information in the display unit 22 
according to the differential image information that is sent from the host PC 1. 

In this manner, the host PC 1 which is located in a distant place can be 
20 operated from the cUent PC 2, and the same image information as in the 
display unit 12 of the host PC 1 can be shown in the display unit 22 of the cUent 
PC 2. 

In this remote control system, if the delay time in the network is large, 
the following phenomena may occur. 
25 i) If the double cHck which is entered in the cUent PC 2 is transmitted 

to the host PC 1 as operation information, it is not processed as a double click in 
the host PC 1. 
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ii) Two single clicks which are entered in the cHent PC 2 are processed 
as a double click at the host PC 1. 

Such incorrect processes may occur if the time interval between packets 
is changed by an irregular delay characteristic of the network while the data of 
5 two packets showing two successive presses on the button of the pointing device 
9 flow on the network. At this time, the host PC 1 recognizes and processes as 
if the button was pressed at a different timing from the operation in 
transmission. 

Moreover, as compared with the case of the user manipulating the host 
10 PC 1 directly, when the host PC 1 is remote-controled by the cHent PC 2, it 
takes time until the input result at the client PC 2 is reflected in the first 
display unit 12 of the host side. Therefore, the user of the cHent computer 2 
takes time until knowing that the double chck is correctly entered at the host 
PC 1. As a result, working efficiency decreases. 

15 

SUMMARY OF THE INVENTION 

It is therefore an object of the invention to present a method, system 
and a computer program product for processing remote control data even if the 
delay time in the network is large. In the present invention, the host computer 

20 terminal and cUent computer terminal are coupled through the network, and 
the remote control system is built up. 

The chent computer terminal comprises an information processing unit, 
a communication unit for exchanging information with the network, and a 
pointing device for controlHng the cUent computer terminal. 

25 In this chent computer terminal, the information processing unit 

transforms the operating procedure of the pointing device into the operation 
information including the button state data, coordinates data, and time interval 
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data between the operations, and the communication unit sends the 
information to the network. 

On the other hand, the host computer terminal comprises an 
information processing unit and a communication unit which are coupled with 
5 the network. In the host computer terminal, the information processing unit 
decomposes the operation information that is received in its communication 
unit into the operating procedure at the cHent terminal, and reproduces the 
operating procedure. 

In this manner, the operating procedure of the pointing device at the 
10 client terminal can be reproduced at the host terminal without having the 
effects of an irregular delay of the network. 



BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 shows a configuration of a remote control system in an 
15 embodiment of the present invention. 

Fig. 2A is a functional block diagram of an information processing unit 
of a cHent PC in the embodiment of the present invention. 

Fig. 2B is a functional block diagram of an information processing unit 
of a host PC in the embodiment of the present invention. 
20 Fig. 3 is a flowchart showing the first button ON judging process in a 

double cHck input of the cHent PC in the embodiment of the present invention. 

Fig. 4 is a flowchart showing the first button OFF judging process after 
the first button is chcked ON in a double chck input of the cHent PC in the 
embodiment of the present invention. 
25 Fig. 5 is a flowchart showing the second button ON judging process 

after the first button is chcked OFF in a double cHck input of the cHent PC in 
the embodiment of the present invention. 

1 
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Fig. 6 is a flowchart showing the process until advancing to the first 
button ON judging process in a double click input after the double click input or 
after going out of the double click input condition of the client PC in the 
embodiment of the present invention. 
5 Fig. 7 is a flowchart showing a reception process of an operation packet 

related to a pointing device of the client PC in the host PC in the embodiment of 
the present invention. 

Fig. 8 shows a configuration of a conventional remote control system. 

10 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT OF THE 

INVENTION 

An embodiment of the present invention is explained with reference to 
Figs. 1 to 7. 

A configuration of the remote control system in the embodiment is 
15 explained with reference to Fig. 1 and Fig. 2. 

Fig. 3 to Fig. 6 are flowcharts showing the transmission process of an 
operation packet about a pointing device 44 of the chent computer in the 
embodiment. 

In Fig. 1, information processing units 31, 41 monitor the operation of 
20 pointing devices 34, 44 of a host computer (host PC) 30 and a chent computer 
(cHent PC) 40, respectively. 

As the pointing device is manipulated, when the indicated coordinates 
are moved or the on/off of the button is changed, the processing units 31, 41 
generate data, and display the coordinates data and the button status data on 
25 the screen of the display units 32, 42 as a pointer, respectively. The processing 
units 31, 41 form data into packets, and send the data to the counterpart 
information processing unit 31, 41 of the partner of communication through 
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communication units 33, 43 and a network 70. 

When the button of the pointing device is turned on, off and on within 
a maximum cHck interval Tl without a change in the coordinates that are 
indicated by the pointing device, the processing units 31, 41 judge an input of 
5 double cUck. 

The processing units 31, 41 have a timer for measuring the maximum 
cHck interval Tl. 

The timer has a function like a watchdog timer, and measures the time 
from a first cUck to a second click on the pointing device. The timer setting 

10 time Tl is counted from 0 seconds, and is the duration from the start of an 
operation at an arbitrary timing until the end of the operation. 

Fig. 2A is a functional block diagram showing the operation of the 
information processing unit 41 of the cUent PC 40. 

Fig. 2B is a functional block diagram showing the operation of the 

15 information processing unit 31 of the host PC 30. 

In Fig. 2A, a timer 60 counts the time as the reference for detecting the 
operation of the pointing device 44. An operation determining unit 62 detects 
and judges the move of the pointing device 44 and an on/off operation of the 
button of the pointing device 44. Based on a result of this detection and 

20 judgement, the operation determining unit 62 further generates operation 
information. This operation information includes the coordinates data 
showing the move of the pointing device 44, button status data showing the 
on/off operation of the button, and the time between the operations. An 
informing unit 64 outputs data for informing the user of the operation of the 

25 pointing device 44. A transmission data output unit 66 outputs the data 
regarding the operation of the pointing device, including the operation 
information, to the network 70 through the communication unit 43. 
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In Fig. 2B, a timer 50 counts the time for judging a double click of the 
pointing device 34 at the host PC 30. A watching unit 52 monitors the 
reception of data regarding the operation of the pointing device 44 including the 
operation information that is sent from the client PC 40. A data determining 
5 unit 54 judges the operation of the pointing device 44 at the client PC 40 based 
on the data including the operation information that is sent from the cUent PC 
40. A reproducing unit 56 reproduces the operation of the pointing device 44 at 
the cUent PC 40 depending on the data that is judged by the data determining 
unit 54 at the host PC 30. 
10 The operation in the client PC 40 will now be explained with reference 

to Figs. 3-6. 

The remote control data processing in the cHent PC 40 includes the 
following processing in the input of a double cHck. 

l) Judging the first button to be ON, 
15 2) Judging the first button to be OFF after judging the first button to 

be ON, 

3) Judging the second button to be ON after judging the first button to 
be OFF, and 

4) Processing before advancing to the first button ON judging process 
20 in a double click input, after the double cHck input or after going out of the 

double cHck input condition. 

The procedure will now be explained sequentially from the flowchart in 

Fig. 3. 

Fig. 3 is a flowchart showing the first button ON judging process in a 
25 double cHck input in the cHent PC 40. 

In Fig. 3, at the click interval acquiring step SlOl, the timer 60 acquires 
the maximum cUck interval Tl for judging a double cUck from the host PC 30. 
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At the first watching step S102, the operation determining unit 62 
judges whether or not the chent side pointing device 44 has been operated. If 
the pointing device 44 has not been operated, the watching step continues until 
the operation is recognized. When the pointing device 44 is operated, the 
5 process goes to the first move judging step S103. 

At step S103, the operation determining unit 62 judges whether or not 
the operation of the pointing device 44 is accompanied by a move. If the 
pointing device 44 has not been moved, it is judged that the button of the 
pointing device 44 is turned on, and the process goes to the first timer start step 
10 S104. If the pointing device 44 has been moved, the process goes to the first 
operation packet transmit step S105. 

At step S104, the timer 60 starts, and the flowchart shown in Fig. 4 is 

started. 

At step S105, the transmission data output unit 66 sends a coordinates 
15 data packet showing the move operation that is generated in the operation 
determining unit 62 to the host PC 30. After transmission of the coordinates 
data packet, the process goes to the first ON judging step S106. 

At step S106, the determining unit 62 judges whether or not the button 
of the pointing device 44 has been turned on. 
20 If the button is turned on, the flowchart shown in Fig. 6 is started. 

If the button is not turned on, going back to step S102, the same process 
is repeated. 

The flowchart shown in Fig. 4 will now be explained. 
Fig. 4 is a flowchart showing the first button OFF judging process after 
25 the first button is judged to be ON in a double chck input in the chent computer 
40. 

At the first lapse time measuring step S201, the timer 60 measures the 
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lapse time, and judges if the time has exceeded Tl or not. When the lapse time 
exceeds Tl, the process goes to the first timer stop step S204. Otherwise, the 
process goes to the second watching step S202. 

At step 202, the determining unit 62 judges whether or not the pointing 
5 device 44 is operated. If the pointing device 44 is judged not to be operated, 
going back to step S201, the timer continues to watch until the lapse time 
exceeds time Tl. 

When the pointing device is judged to be operated, the process goes to 
the second move judging step S203. 
10 At step S203, it is judged if the operation of the pointing device 44 is 

accompanied by a move or not. 

If the pointing device 44 is not accompanied by a moved, the 
determining unit 62 judges that the device 44 is turned off, and the flowchart in 
Fig. 5 is started. 

15 On the other hand, if the pointing device 44 is judged to have been 

accompanied by a moved, it is regarded that the pointing device 44 is moved 
with the button remaining in the ON position, and the process goes to the 
second timer stop step S206. 

At step S206, the timer 60 stops, and the transmission data output unit 
20 66 transmits the packet data showing the button ON to the host PC 30 through 
the communication unit 43 (the second ON packet transmit step S207). 

In succession, the coordinates data packet of the pointing device 44 is 
sent to the host PC 30 (the second operation packet send step S208). 

After transmission of the coordinates data packet of the pointing device 
25 44, the process goes to the first OFF judging step S209. 

At step S209, the determining unit 62 judges whether or not the ON 
state button is manipulated to the OFF state. 
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If the button is manipulated to the OFF state, going back to step S102 
in Fig. 1, the same process is repeated. 

When the button remains in the ON position, the flowchart in Fig. 6 is 

started. 

5 At the first timer stop step S204, the timer 60 stops, and the process 

goes to first ON packet send step S205. 

At step S205, the output unit 66 transmits the button ON packet data 
to the host PC 30, and the flowchart in Fig. 6 is started. 

The steps shown in the flowchart in Fig. 5 will now be explained. 
10 Fig. 5 is a flowchart showing the second button ON judging process 

afl:er judging the first button to be OFF in a double cHck input in the client PC 
40. 

At the second lapse time measuring step S301, the timer 60 measures 
the lapse time, and judges whether or not the time has exceeded Tl. When the 
15 lapse time exceeds Tl, the process goes to the fourth timer stop step S307. 
Otherwise, the process goes to the third watching step S302. 

At step 302, the determining unit 62 judges whether or not the second 
pointing device 44 is operated. If the pointing device 44 is not operated, going 
back to step S301, the timer continues to watch until the lapse time exceeds 
20 time Tl. When operated, the process goes to the third timer stop step S303. 

At step S303, the timer 60 stops, and the process goes to the third move 
judging step S304. 

At step S304, the determining unit 62 judges whether or not the 
operation of the pointing device 44 of the cUent PC 40 is accompanied by a 
25 move. 

If the pointing device 44 has not been accompanied by a move, the 
determining unit 62 judges that the device 44 is turned off. At this time, the 
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output unit 66 transmits an operation command showing a double click to the 
host PC 30 (double click send step S305). 

Consequently, the informing unit 64 transmits data to an audio output 
unit or a video output unit of the client PC 40 so as to inform the user of the 
5 transmission of a double cHck command by voice or display (informing step 
S306). 

The next process goes to step S401 in Fig. 6. 

If the pointing device 44 is moved by an operation, on the other hand, 
the determining unit 62 judges that the button is turned on (step S102), and 
10 then turned off (step S202), and consequently that the pointing device 44 is 
moved. Therefore, the transmission data output unit 66 first transmits packet 
data showing the button to be ON to the host PC 30 (fourth ON packet send 
step S310). 

Next, the transmission data output unit 66 transmits packet data 
15 showing the button to be OFF to the host computer 30 (fourth OFF packet send 
step S311). 

Further, the transmission data output unit 66 transmits the coordinates 
data showing the coordinates of the moved position of the pointing device 44 to 
the host computer 30 (third operation packet send step S3 12). Then, the 
20 process goes to the third ON judging step S313. 

At the second lapse time measuring step S301, when the lapse time of 
the timer 60 exceeds Tl, the process goes to the fourth timer stop step S307. 

At step S307, the timer 60 stops, and the process goes to the third ON 
packet transmit step S308. 
25 At step S308, the transmission data output unit 66 transmits the 

packet data showing a button to be ON to the host PC 30, and the process goes 
to third OFF packet send step S309. 
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At step S309, the transmission data output unit 66 transmits the 
packet data showing a button to be OFF to the host PC 30, and the process 
returns to step S102 in Fig. 1. 

At the third ON judging step S3 13, the operation determining unit 62 
5 judges whether or not the operation judged at step S302 is an ON operation of 
the button. 

In the case of an ON operation of the button, the process goes to the 
fourth watching step S401 in Fig. 6. 

Otherwise, the process returns to step S102 in Fig. 1. 
10 The flowchart shown in Fig. 6 will now be explained. 

Fig. 6 is a flowchart showing the process until advancing to the first 
button ON judging process in a double click input after the double cKck input or 
after going out of the double cUck input condition in the cUent PC 40. 

At the fourth watching step S401, the determining unit 62 judges 
15 whether or not the second pointing device 44 is operated. If the pointing 
device 44 is not operated, watching continues until an operation of the pointing 
device 44 is recognized. 

When the pointing device 44 is operated, the process goes to the fourth 
operation packet send step S402. 
20 At step S402, the transmission data output unit 66 transmits the 

operation packet showing the operation judged at step S401 to the host 
computer 1, and the process goes to the fourth OFF judging step S403. 

At step S403, the determining unit 62 judges whether or not the 
operation that is judged at step S401 is an OFF operation of the button. 
25 In the case of the OFF operation of the button, the process returns to 

step S102 in Fig. 1. 

Otherwise, the process returns to step S401. 
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Thus, in the information processing unit 41 of the cHent PC 40, the 
operation determining unit 62 judges the operation of the pointing device 44, 
and generates operation information as packet data. The data output unit 66 
outputs the operation information as packet data to the network 70 through the 
5 communication unit 43. 

The processing of remote control data in the host PC 30 will now be 
explained. 

Fig. 7 is a flowchart showing the reception process of an operation 
packet, that is, operation information regarding the pointing device 44 of the 
10 client PC 40 in the host PC 30 in the embodiment. 

Herein, T2 is the maximum cUck interval for judging a double cHck by 
the ON-OFF-ON button operation of the pointing device 34 in the host PC 30. 
The timer 50 stops automatically when exceeding the time T2. 

In Fig. 7, at a reception watching step S501, the watching unit 52 
15 judges whether or not the operation packet data that is transmitted from the 
cUent computer 40 is received. 

If the operation packet data is not received, the watching unit 52 
continues to monitor until the reception of the operation packet data is 
confirmed. 

20 When the operation packet data is received, the process goes to the 

fourth move judging step S502. 

At step S502, the data determining unit 54 judges whether or not the 

operation packet data that is received at step S501 shows the operation as 

being accompanied by a move of the pointing device 44 of the coordinates. 
25 If accompanied by a move of the coordinates, the process goes to the 

fourth timer stop step S503. At this step, the timer 50 stops, and the process 

goes to the double cUck judging step S504. 
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If not accompanied by a move, the process goes to step S504. 

At step S504, the data determining unit 54 judges whether or not the 
operation packet data that is received at step S501 shows a double chck. 

If the received operation packet data shows a double cUck, the process 
5 goes to the first timer operation watching step S505. 

If the received operation packet data does not show a double chck, the 
process goes to the button operation judging step S509. 

At the first timer operation watching step S505, the watching unit 52 
judges whether or not the timer 50 is in operation. If the timer 50 is in 
10 operation, the watching unit 52 waits until stopping automatically. Then, the 
reproducing unit 56, after the first ON reproducing step S506, the OFF 
reproducing step S507, and the second ON reproducing step S508, reproduces 
the ON'OFF-ON operation of the button of the pointing device 44 so as to be 
recognized as a double click in the host PC 30. 
15 If judged not to be a double chck at step S504, the process goes to the 

button operation judging step S509. 

At step S501, the data determining unit 54 checks whether or not the 
received operation packet shows the operation of changing the button of the 
pointing device 44 fi:om ON to OFF. If Yes, the process goes to the second 
20 timer operation watching step S510. Otherwise, the process goes to the 
operation packet reproducing step S512 for reproducing the button operation. 

At the second timer watching step S510, the data determining unit 54 
judges whether or not the timer 50 is in operation. During operation, waiting 
until stopping automatically, the process goes to the second timer start step 
25 S511. 

At step S511, the timer 50 starts, and the process goes to the operation 
packet reproducing step S512. 
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At step S512, the reproducing unit 56 reproduces the operation that is 
indicated by the operation packet received at step S501. 

Thus, in the information processing unit 31, the data determining unit 
54 in the host PC 30 judges the operation of the pointing device 44 in the client 
5 PC 40 by the data including the operation information that is sent from the 
chent PC 40. The reproducing unit 56 reproduces the operation of the pointing 
device 44 in the chent PC at the host PC 30 depending on the data that is 
judged by the data determining unit 54. 

According to the embodiment of the present invention, the double cHck 

10 is transmitted from the client computer to the host computer, not as plural 
operation packets showing an on/off operation of the button of the pointing 
device of the cUent computer, but instead as one operation packet showing the 
double chck. Therefore, even if the delay time of the network is large, the host 
computer correctly recognizes the input of a double cUck at the chent computer. 

15 Moreover, the user of the client computer, by voice or display, is notified 

of the transmission of an operation packet showing a double chck input from 
the chent computer to the host computer. Thus, the user instantly 
understands the input of a double chck. 

As described herein, in the remote control data processing method and 

20 system of the present invention, the information processing unit of the cUent 
computer converts the operating procedure of the pointing device of the chent 
computer into operation information including button status data, coordinates 
data, and interval data between operations, and transmits the operation 
information to the host computer. In the information processing unit of the 

25 host computer, the operating procedure of the chent computer is reproduced 
from the received operation information. Thus, even if the delay time of the 
network is large, the host computer correctly recognizes the operation that is 
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entered in the client computer. 

In the foregoing embodiment, the specific method and device are 
explained. The present invention may also be reahzed by using a computer 
program product containing a computer program for executing the method 
shown in the embodiment by a computer. 
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